Compared to conventional data storage devices such as hard disk drives, solid-state drives (SSDs) provide more complexity and pseudo-general programmability. For example, flash memory-based SSDs increasingly package more powerful CPUs and a higher-capacity memory and storage and make those resources available to a user application. Moreover, for data intensive applications, it is considered that moving codes to data is far more efficient than moving data to codes. Thus, efficient data processing systems try to push codes as far down into a query-processing pipeline as possible by using techniques such as early selection pushdown and early aggregation (or pre-aggregation). Similarly, parallel or distributed data processing systems can run as much of the queries close to a node that holds the data.
Traditionally, code-offloading techniques have been implemented by tweaking an operating system (OS) of a host computer and a firmware of an SSD in an ad-hoc manner. Although the boundaries among persistent storage, volatile storage, and processing are increasingly getting blurrier, the existing functional separation among the host OS, libraries, fixed protocols, and a storage device abstraction layer commonly introduces redundant implementations and can become a bottleneck or a wasteful resource-consumer.
Non SQL (NoSQL) databases (DBs) (e.g., key-value stores, document stores, graph, object databases, etc.) are widely used in modern computer systems. Compared to relational DBs, NoSQL DBs are simple, flexible and lightweight and can provide excellent scalability and large performance gains with certain workloads. The rapid move to cloud computing and large systems with big data contributes to the growing popularity of NoSQL DBs.
FIG. 1 shows a traditional NoSQL database (DB) system. The NoSQL DB system 100 includes overlapped components and redundant mappings. The NoSQL DB system 100 includes a host computer 110 and a storage device 150 communicating over a host-to-SSD interface 190. The host computer 110 includes a NoSQL DB 120 that has a mapping table for key to LBA translation 121, an operating system 130, and an SSD interface 140. The examples of the host-to-SSD interface 190 include, but are not limited to, Peripheral Component Interconnect Express (PCIe), Serial Attached SCSI (SAS), Ethernet, and Serial AT Attachment (SATA). The storage device 150 includes a memory controller 160 and a non-volatile memory 170. The memory controller 160 includes a host interface 161, a LBA-PBA mapping table 162, an error-correcting code (ECC) memory 163, and a flash interface 164 for interfacing with the non-volatile memory 170. The host computer 110 and the storage device 150 communicate with commands, LBAs, and data over the host-to-SSD interface 190.
There are two redundant mapping layers including a first mapping layer for key-LBA mapping and a second mapping layer for LBA-PBA mapping. Traditionally, the host computer 110 and the storage device 150 communicate with each other based on the LBAs to specify the address of storage locations in the storage device 150. For compatibility, the storage device 150 works with LBAs, and a flash translation layer (FTL) of the memory controller 160 maintains the LBA-PBA translation. The FTL translates logical block addresses (LBAs) into physical block addresses (PBAs) using the mapping table 162, and accesses the physical blocks of the non-volatile memory 170 as indicated by the mapping table 162 to perform the requested operations (e.g., read, write, read/modify/write (RMW), and delete) on the physical blocks of the non-volatile memory 170.
The key-LBA mapping in the host computer 110 can be eliminated if proper arrangements are made between a host computer and a storage device. FIG. 2 shows an example of a system architecture for another traditional NoSQL DB system. The NoSQL DB system 200 can include a host computer 210 and a storage device 250 communicating over a host-to-SSD interface 290. The host computer 210 can include a NoSQL user application program interface (API) 220, an operation system 230, and an SSD interface 240. Similar to the host-to-SSD interface 190 of FIG. 1, the host-to-SSD interface 290 can be any of PCIe, SAS, Ethernet, and SATA, or the like. The storage device 250 can include a memory controller 260 for interfacing with a non-volatile memory 270. The memory controller 260 can include a host interface 261, a NoSQL DB integrated FTL 262, an error-correcting code (ECC) memory 263, and a flash interface 264. The NoSQL DB integrated FTL 262 can eliminate the key-LBA mapping in the host computer 210, therefore the NoSQL DB system 200 can have a better aggregated performance compared to the NoSQL DB system 100 of FIG. 1 that requires two redundant mapping layers.
The host computer 210 and the storage device 250 can communicate with query commands, keys, and data over the host-to-SSD interface 290. The non-volatile memory 270 can be of various types of non-volatile memory including, but not limited to, a flash memory, a phase-change RAM (PRAM), a spin-transfer torque magnetic random access memory (STT-MRAM), and a resistive RAM (ReRAM).
A firmware of the present polymorphic storage device defining behaviors of the storage device can be reconfigured via a firmware update. Through the firmware update, the present polymorphic storage device can be transformed into a different type of devices from the original configuration. For example, the present polymorphic storage is originally configured as a general-purpose storage device. By a firmware update, it can be converted into a special-purpose device, such as an in-storage computing device, a key-value store device, a Hadoop distributed file system (HDFS) device, an object-store device, etc.
FIG. 3 shows layers of a memory controller of an example storage device. The memory controller 360 has four layers, namely, a host interface layer (HIL) 310, a flash translation layer (FTL) 320, a virtual flash layer (VFL) 330, and a flash interface layer (FIL) 340. These four layers can be implemented in the firmware of the storage device that can run on one or more embedded processors of the storage device. The host interface layer 310 can implement a bus interface protocol such as SATA, SAS, or PCI Express (PCIe). The flash translation layer 320 is responsible for mapping logical block addresses (LBAs) in the host OS to physical block addresses (PBAs) in a non-volatile memory of the storage device. The virtual flash layer 330 and the flash interface layer 340 are responsible for data transfer between the non-volatile memory and a DRAM memory of the storage device. The firmware of the storage device can provide other features including an ECC logic and a direct memory access (DMA). To enhance the I/O performance of the non-volatile memory, the memory controller 360 can use interleaving techniques at a chip-level and/or a channel-level.